Add file storage option #12590

nucleogenesis · 2024-08-22T21:23:54Z

Summary

In order to allow us to use a cloud backend for the File Storage in Django, this adds a value to options.py which defaults to the Django FileSystemStorage (which... it did anyway but now we make sure of it).

This should make it configurable on BCK such that if there is a module that is a Google cloud backend class that implements the Django Storage class, then it can be added as the value for the settings.

For example, if we have a new class "GCloudStorage" in kolibri.core.storage then we would use that class if we set the option added here to kolibri.core.storage.GCloudStorage.

This is very much a first whack -- one thing I'm not clear on is if by naming the option by the name that Django would look to in the env vars DEFAULT_FILE_STORAGE does the Kolibri options.py stuff automatically apply that setting because of the matching name?

References

Fixes #9441 (or at least begins to address it)

Reviewer guidance

Run Kolibri without changing the options.ini file, do a file storagey thing (I generate a Facility Data CSV)
Uncomment the STORAGE_BACKEND and change it's value to gcs and restart Kolibri, you should get an error message that the module is not available
pip install -r requirements/storages.txt then try again, no error and it should start up

Next steps

Work out a way to test this in a BCK environment - will chat to @jredrejo for guidance from his experience w/ KDP.
Test it that way too :)

Testing checklist

Contributor has fully tested the PR manually
If there are any front-end changes, before/after screenshots are included
Critical user journeys are covered by Gherkin stories
Critical and brittle code paths are covered by unit tests

PR process

PR has the correct target branch and milestone
PR has 'needs review' or 'work-in-progress' label
If PR is ready for review, a reviewer has been added. (Don't use 'Assignees')
If this is an important user-facing change, PR or related issue has a 'changelog' label
If this includes an internal dependency change, a link to the diff is provided

Reviewer checklist

PR is fully functional
PR has been tested for accessibility regressions
External dependency files were updated if necessary (yarn and pip)
Documentation is updated
Contributor is in AUTHORS.md

github-actions · 2024-08-22T21:54:21Z

Build Artifacts

Asset type	Download link
PEX file	kolibri-0.18.0a0.dev0_git.288.gd76e213d.pex
Windows Installer (EXE)	kolibri-0.18.0a0.dev0+git.288.gd76e213d-windows-setup-unsigned.exe
Debian Package	kolibri_0.18.0a0.dev0+git.288.gd76e213d-0ubuntu1_all.deb
Mac Installer (DMG)	kolibri-0.18.0a0.dev0+git.288.gd76e213d.dmg
Android Package (APK)	kolibri-0.18.0a0.dev0+git.288.gd76e213d-0.1.4-debug.apk
Raspberry Pi Image	kolibri-pi-image-0.18.0a0.dev0+git.288.gd76e213d.zip
TAR file	kolibri-0.18.0a0.dev0+git.288.gd76e213d.tar.gz
WHL file	kolibri-0.18.0a0.dev0+git.288.gd76e213d-py2.py3-none-any.whl

rtibbles

I think we probably want to keep the options.py interface restricted, others can override settings if they see fit, but we should focus on enabling the specific options we need.

Hint: if you do kolibri configure list-env you will see a complete list of available env vars for configuration (which will also show how your new option can be set as an env var).

rtibbles · 2024-08-23T00:30:03Z

kolibri/utils/options.py

@@ -359,6 +377,16 @@ def lazy_import_callback_list(value):


 base_option_spec = {
+    "FileStorage": {
+        "DEFAULT_FILE_STORAGE": {


I think I'd want to hew closer to the pattern we have for the Cache and Database options here - and offer simple, pre-specified string options that refer to specific backends. If someone really wants to run a custom backend, they can override the settings file and do what they like.

That way, with specific backends in mind, we can then explicitly enumerate the additional things that need to be specified in each case - for example the default "file_system" backend value will need a path, if it's a GCS backend then other things might be required (or may be automagically configured in some cases).

rtibbles · 2024-08-23T00:30:38Z

kolibri/utils/options.py

+    except ImportError:
+        logger.error("Default file storage is not available.")
+        raise VdtValueError(value)
+    except Exception:


Shouldn't ever be catching a bare Exception, unless for very good reason - it can hide a multitude of sins.

rtibbles · 2024-08-23T00:31:51Z

kolibri/utils/options.py

+        modules = value.split(".")
+        klass = modules.pop()
+        module_path = ".".join(modules)
+        module = importlib.import_module(module_path)


Note that Django exposes a utility called import_string which we already use in this module for loading classes by a string dot path, so this seems preferable to use here.

nucleogenesis · 2024-08-23T19:59:15Z

kolibri/utils/options.py

@@ -15,6 +15,7 @@
 from configobj import ConfigObj
 from configobj import flatten_errors
 from configobj import get_extra_values
+from django.core.files.storage import Storage


This is probably why flake8 wouldn't let pre-commit pass but it didn't give me useful output

jredrejo · 2024-08-27T16:10:47Z

kolibri/utils/options.py

@@ -737,6 +766,7 @@ def _get_validator():
            "url_prefix": url_prefix,
            "bytes": validate_bytes,
            "multiprocess_bool": multiprocess_bool,
+            "storage_option": storage_option,


how will this work with database based cache?

Hm - I'm not sure. I assumed this was only related to validation on initialization

nucleogenesis · 2024-12-20T23:05:10Z

kolibri/core/auth/management/commands/bulkexportusers.py

+            logger.info("File saved - Path: {}".format(file_storage.url(file)))
+            logger.info("File saved - Size: {}".format(file_storage.size(file)))


@jredrejo I've tried several things around here, but no matter what I do I the file always shows size 0... I've confirmed that there are users (usernames is full of data, for example) -- so I'm not sure why the writer isn't updating the file object here...

The problem was not in reading the file info, but writing it. It had 0 bytes. I have made a commit in this PR to ensure data is flushed into the BytesIO and now it's working:

INFO 2024-12-26 20:45:45,313 Invoking command bulkexportusers INFO 2024-12-26 20:45:45,373 Creating users csv file /home/jose/.kolibri/log_export/Kolibri en casa de admin_3da4_users.csv [#######################################################################################################################################################################################-------------------------------------------------------------] 75%INFO 2024-12-26 20:45:46,767 File saved - Path: https://storage.googleapis.com/kdp-csv-reporting-develop/Kolibri_en_casa_de_admin_3da4_users.csv INFO 2024-12-26 20:45:47,073 File saved - Size: 482 INFO 2024-12-26 20:45:47,073 Created csv file /home/jose/.kolibri/log_export/Kolibri en casa de admin_3da4_users.csv with 4 lines

(I've also rebased develop in the branch because there was already a conflict and more might appear soon)

…FAULT_FILE_STORAGE

…underlying BytesIO buffer

nucleogenesis requested a review from rtibbles August 22, 2024 21:23

github-actions bot added the DEV: backend Python, databases, networking, filesystem... label Aug 22, 2024

rtibbles reviewed Aug 23, 2024

View reviewed changes

nucleogenesis commented Aug 23, 2024

View reviewed changes

nucleogenesis requested a review from jredrejo August 26, 2024 23:13

rtibbles self-assigned this Aug 27, 2024

rtibbles added this to the Kolibri 0.17: Planned Patch 1 milestone Aug 27, 2024

jredrejo reviewed Aug 27, 2024

View reviewed changes

rtibbles modified the milestones: Kolibri 0.17: Planned Patch 1, Kolibri 0.17: Planned Patch 2 Sep 16, 2024

rtibbles modified the milestones: Kolibri 0.17: Planned Patch 2, Kolibri 0.18: General maintenance Oct 22, 2024

github-actions bot added the APP: Facility Re: Facility App (user/class management, facility settings, csv import/export, etc.) label Dec 20, 2024

nucleogenesis commented Dec 20, 2024

View reviewed changes

nucleogenesis and others added 11 commits December 26, 2024 17:59

add file storage option, validator

e348a34

WIP

ee61e61

validate storage option; add reqs/storages.txt

fced765

return the value from gcs storage_option validation fn

250558f

simplify imports, add filestorage util class to pass importpath to DE…

7c44493

…FAULT_FILE_STORAGE

exportusers & csv_utils updated to use DefaultStorage

b792ce9

simplify gcs settings and calsses

237d657

update facility views to use DefaultStorage in csv file response

6be0f40

logs csv_export use DefaultStorage

f5d0d66

WIP - Users CSV export use DefaultStorage

e052bb4

flushing to ensure all data from the TextIOWrapper is written to the …

953b29b

…underlying BytesIO buffer

jredrejo force-pushed the fix--dynamic-file-storage-backend-cloud-file-storage branch from 43309bc to 953b29b Compare December 26, 2024 19:48

github-actions bot added DEV: dev-ops Continuous integration & deployment DEV: renderers HTML5 apps, videos, exercises, etc. APP: Device Re: Device App (content import/export, facility-syncing, user permissions, etc.) labels Dec 26, 2024

[pre-commit.ci lite] apply automatic fixes

38368cb

jredrejo changed the base branch from release-v0.17.x to develop December 26, 2024 20:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add file storage option #12590

Add file storage option #12590

nucleogenesis commented Aug 22, 2024 •

edited

Loading

github-actions bot commented Aug 22, 2024 •

edited

Loading

rtibbles left a comment

rtibbles Aug 23, 2024

rtibbles Aug 23, 2024

rtibbles Aug 23, 2024

nucleogenesis Aug 23, 2024

jredrejo Aug 27, 2024

nucleogenesis Aug 28, 2024

nucleogenesis Dec 20, 2024

jredrejo Dec 26, 2024

		logger.info("File saved - Path: {}".format(file_storage.url(file)))
		logger.info("File saved - Size: {}".format(file_storage.size(file)))

Add file storage option #12590

Are you sure you want to change the base?

Add file storage option #12590

Conversation

nucleogenesis commented Aug 22, 2024 • edited Loading

Summary

References

Reviewer guidance

Testing checklist

PR process

Reviewer checklist

github-actions bot commented Aug 22, 2024 • edited Loading

rtibbles left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nucleogenesis commented Aug 22, 2024 •

edited

Loading

github-actions bot commented Aug 22, 2024 •

edited

Loading